<html>
<head><meta charset="utf-8"><title>cargo vendor -Zbuild-std · t-cargo · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/index.html">t-cargo</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html">cargo vendor -Zbuild-std</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="215093116"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215093116" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215093116">(Oct 30 2020 at 13:21)</a>:</h4>
<p>I'm currently working on the problem of getting <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1671691">tsan-instrumented builds of std in firefox's CI</a>. The current recommended way to do this is <a href="https://doc.rust-lang.org/cargo/reference/unstable.html#build-std">with <code>cargo -Zbuild-std</code></a>. However, Firefox's CI is a vendored environment, so the dependencies of std need to be pre-resolved and vendored for this to work. I could potentially just work out by hand what the missing vendors are, but I'd like an <em>easily maintainable</em> solution. One that doesn't make compiler version bumps a huge nightmare. I briefly considered <a href="https://github.com/rust-lang/rust/issues/78533">pre-built instrumented binaries</a> but this requires me to reinvent cargo's logic for not using the instrumented std for build scripts and plugins, and seemed like a big project to simulacrum.</p>
<p>I am now persuing the avenue of thought: maybe <code>cargo vendor -Zbuild-std</code> should Just Work. Clearly cargo has the ability to pull up the Cargo.toml for std and compute its build graph. If we could teach <code>cargo vendor</code> to incorporate that information, my problem would be perfectly solved!!!(?) If this sounds reasonable, I'm willing to help / do all the work for the implementation (though would need a bit of guidance on what to change).</p>



<a name="215093746"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215093746" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215093746">(Oct 30 2020 at 13:27)</a>:</h4>
<p>NB: I haven't fully thought out the implications of this feature with people showing up with their own compiler toolchains. Perhaps it's something the CI would do Just-In-Time. Perhaps it's just something a careful person would have to do explicitly with exactly the right toolchain locally installed.</p>



<a name="215094322"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215094322" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215094322">(Oct 30 2020 at 13:32)</a>:</h4>
<p>Oh additional context: there's the <a href="https://static.rust-lang.org/dist/2020-03-25/rustc-nightly-src.tar.xz">big rustc vendor tarball</a> which we could potentially use to solve this problem, but I'm not sure how big of a problem it's large size / huge pile of deps not needed by std would be. Need to discuss this with team, but currently working on the assumption that this isn't the path we want to take.</p>



<a name="215094848"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215094848" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215094848">(Oct 30 2020 at 13:37)</a>:</h4>
<p>Additional Additional Concern: ideally folks would still be able to do normal <code>cargo vendor</code> updates to the tree, without worrying about the <code>-Zbuild-std</code> usecase (because it's a CI thing, and they won't necessarily have the CI toolchain installed). Perhaps this pushes more to the "CI will run a build-std vendor" solution. But I could also imagine some system where the std deps are specially identified and able to not be clobbered by other non-build-std vendors. Sounds messy though...</p>



<a name="215095268"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215095268" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215095268">(Oct 30 2020 at 13:41)</a>:</h4>
<p>That said, we already <a href="https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/vendor/vendor_rust.py">have a wrapper script</a> that guards and rewrites the vendor results, so it wouldn't be completely wild for us to somehow do the special build-std preservation ourselves if it doesn't make sense in cargo</p>



<a name="215095463"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215095463" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215095463">(Oct 30 2020 at 13:42)</a>:</h4>
<p>(maybe generate a dummy library in our workspace with the deps that std reported, so future vendors incorporate it?)</p>



<a name="215112165"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215112165" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215112165">(Oct 30 2020 at 15:44)</a>:</h4>
<p><code>cargo vendor -s &lt;path-to-rust-src-std-Cargo.toml&gt;</code> seems to <em>almost</em> work, but it doesn't seem to consult the rust Cargo. lock, while -Zbuild-std does, so e.g. vendor pulls down <code>compiler_builtins 0.1.36</code> but the lock enforces <code>=0.1.35</code> so the build fails</p>



<a name="215114679"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215114679" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215114679">(Oct 30 2020 at 16:01)</a>:</h4>
<p>Ok if you -s in both <code>std</code> and <code>test</code>, and repeatedly vendor+build+add-a-strict-dep-to-your-own-project-to-fix-the-issue you eventually do get a vendored build. If the lock's constraints could be incorporated automatically, this would be a pretty reasonable thing, I think?</p>



<a name="215115504"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215115504" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215115504">(Oct 30 2020 at 16:07)</a>:</h4>
<p>I would love to see this work in one step.</p>



<a name="215116580"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215116580" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215116580">(Oct 30 2020 at 16:15)</a>:</h4>
<p>Looking at the source for <code>cargo vendor</code> it looks like it might already be trying to incorporate the Cargo.lock for std but it doesn't find it because the rust tree is configured weird...?</p>



<a name="215116646"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215116646" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215116646">(Oct 30 2020 at 16:15)</a>:</h4>
<blockquote>
<p>it doesn't find it because the rust tree is configured weird...?</p>
</blockquote>
<p><del>story of my life</del></p>



<a name="215117509"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215117509" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215117509">(Oct 30 2020 at 16:21)</a>:</h4>
<p>ok notable progress: if you just copy the Cargo.lock in src/rust to std and test, it vendors everything correctly</p>



<a name="215117529"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215117529" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215117529">(Oct 30 2020 at 16:21)</a>:</h4>
<p>so that's hacky but reproducible and reliable</p>



<a name="215118841"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215118841" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215118841">(Oct 30 2020 at 16:31)</a>:</h4>
<p>now as a script with some hardcoding <a href="https://gist.github.com/Gankra/75712d7474ae0a086dd899b091451a2b">https://gist.github.com/Gankra/75712d7474ae0a086dd899b091451a2b</a></p>



<a name="215119213"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215119213" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215119213">(Oct 30 2020 at 16:34)</a>:</h4>
<p>not sure what the Cool Kids use to get the toolchain path</p>



<a name="215119396"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215119396" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215119396">(Oct 30 2020 at 16:36)</a>:</h4>
<p><span class="user-mention" data-user-id="137587">@Gankra</span> <code>rustc --print sysroot</code></p>



<a name="215119926"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215119926" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215119926">(Oct 30 2020 at 16:40)</a>:</h4>
<p>hell yea, that's super simple and clean</p>



<a name="215123993"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215123993" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215123993">(Oct 30 2020 at 17:13)</a>:</h4>
<p>The issues <a href="https://github.com/rust-lang/wg-cargo-std-aware/issues/23">https://github.com/rust-lang/wg-cargo-std-aware/issues/23</a> and <a href="https://github.com/rust-lang/wg-cargo-std-aware/issues/11">https://github.com/rust-lang/wg-cargo-std-aware/issues/11</a> are tracking this. I haven't thought about it too much. We've recently discussed adding all the dependencies to the <code>rust-src</code> component, but I don't know how those would be wired in. I don't think source replacement would work, and the only other option I can think of is have some kind of blanket set of patches, but which crates to patch aren't known until it's too late, so I'm unsure how to solve it.</p>



<a name="215125117"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215125117" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215125117">(Oct 30 2020 at 17:21)</a>:</h4>
<p>Ick, the hack script needs to compensate for an old dir structure too <a href="https://gist.github.com/Gankra/6a025b5cce9d9b047e46a5caeded3050">https://gist.github.com/Gankra/6a025b5cce9d9b047e46a5caeded3050</a></p>



<a name="215126191"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/cargo%20vendor%20-Zbuild-std/near/215126191" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Gankra <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/cargo.20vendor.20-Zbuild-std.html#215126191">(Oct 30 2020 at 17:29)</a>:</h4>
<p><span class="user-mention" data-user-id="120518">@Eric Huss</span> oh thanks for the links -- connected up the tracking issues</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>